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ABSTRACT 

This report introduces the concept of a 
"metacourse" — i.e., the provision of mental models, problem solving 
strategies, key concepts, and other structures — to assist students in 
learning programming, and examines the effect of the use of such a 
metacourse on high school programming students. The experimental 
group consisted of 6 teachers who taught BASIC programming to 132 
students at 5 high school laboratory sites, and the control group 
consisted of 9 teachers who taught 239 students at 8 control sites. 
The metacourse consisted of a series of eight instructional lessons 
interspersed throughout the semester. Students were given both 
pretests and posttests in cognitive skills, and observations of the 
experimental sites were also made. After examining these data, a 
second revised metacourse was developed and used with nine other 
teachers in seven new sites under conditions that more normally 
duplicate normal classroom innovations. The results showed that both 
the laboratory site and classroom site students who were exposed to 
the metacourse experienced significant improvement in their mastery 
of the BASIC programming language. However, the modest evidence of 
"transfer" skills found in the laboratory site was not evident in the 
second study under more normal classroom conditions. This suggests 
that transfer from programming is not normally obtained unless 
special efforts are made to explicitly design such elements into the 
intervention. Appended materials include the cognitive 
pretest/posttest, a test on BASIC, and a classroom observation 
worksheet. (34 references) (EW) 
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Emptrical Studies of a "Metacourse" to 
Enhance the Learning of BASIC 



Introduction 

Enhancing instruction in an existing subject matter on a wide scale is a conspicuous and 
rarely met challenge of contemporary education, and for understandable reasons. On the one 
hand, the subject matters bring with them a number of conceptual challenges and problems of 
metacognitive control that require careful study to disclose end careful instructional design to 
remedy. On the other hand, however artfully fashioned better instruction may be, wide-scale 
implementation must run the gauntlet of a number of impeding factors - teacher training, cost of 
materials, institutional inertia, and so on. 

A desirable program of research would address not only the leaming of a subject matter but 
also the practical problems of implementation as part of a unified program of inquiry. During the 
past several years, we have tried to conduct such an investigation Into the pedagogy of 
programming. Through clinical studies and teaching experiments, we have sought to understand 
better the factors that interfere with beginners* mastery of programming in BASIC and LOGO and 
to devise instructional methods that enhance their leaming (see e.g. Peri<ins, Farady, Hancock, 
Hobbs, Simmons, Tuck, & Villa, 1986; Pertains, Hancock, Hobbs, Martin, & Simmons, 1986; 
Peri<ins, Hancock, Hobbs, Martin, & Simmons, 1986; Peri<ins, Martin, & Farady, 1986). Regarding 
implementaiton, we have tried to couch our instructional methods in a format called a 
"metacourse," designed from the first to address some of the daunting problems of wide-scale 
disseminabillty (Peridns, Farady, Simmons, &Villa; Pjridns, Schwartz, & Simmons, in press). 

This paper describes a large-scale experiment examining the effectiveness of the 
metacourse we have devetoped in enhancing high school students' leaming of BASIC. The 
positive results can be taken as support for the general analysis of the difficulties of programming 
developed in our eariier wori<, and for the viability of our approach to educational change. Before 
detailing the experimental method and results, we describe briefly the context established by our 
prior wori(. 

Students' Leaming Problems in Mastering Programming 

It is plain that computer programming poses special challenges. Like mathematics and 
physfcs, programming is a precision-intensive subject matter, requiring mettoulous care with 
details. Programming is also problem-solving intensive, student activities focusing almost entirely 
on resolving progranwning problems. While today this is typical of mathematics, physics, and 
certain other science subjects, we note that an^ school subject can and periiaps should be 
treated in a problenvsolving intensive way. Rnally, programming is not just problem-solving 
Intensive but "design intensive." That is, students have to constmct whole complex produces that 
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do certain jobs, not just derive particular answers like 35 cm/sec or A»1 7. In today's schools, 
complex products appear nwstly in art (the works themselves), Euclidean geometry (proofs), and 
English and Social Studies (essays). Again, we note that any school subject can and perhaps 
should emphasize the construction of complex products. 

It is hardly surprising that a precision-intensive, problem-solving intensive, and design- 
intensive subject matter should give many students considerable difficulty. In recent years, 
evidence has accumulated that a high percentage of students in elementary and high school 
achieve only extremely limited mastery of programming even after a semester or two of instruction 
(see e.g. Mawby, 1S87; Pea, 1986; Pea & Kuriand, 1984a; Pea & Kuriand, 1984b: Kuriand, Pea, 
Clement, & Mawby, 1986; Kuriand, Clement, Mawby, & Pea, 1987; Eonar & Soloway, 1985; 
Soloway & Ehriich, 1984; Sleeman, Putnam, Baxter, & Kuspa, 1986). In seel<ing to summarize 
and synthesize our own and others' findings in this area, we have found it useful to characterize 
students' difficulties under three broad headings: fragile knowledge, a shortfall in elementary 
problem solving strategies, and problems of confkJence and control. 

Fraoile KnowlPdna 

"Fragile knowledge" refers to the fact that students commonly display partial knowledge, 
considerable inert knowledge (not evoked in contexts of need but retrievable with cueing), and 
gartjied knowledge (concepts used in the wrong place, in inappropriate hybrids) of programming. 
PecWns, Hancock, Hobbs, Martin, and Simmons (1986) discuss fragile programming knowledge in 
detail, emphasizing the importance of distinguishing between fragile and missing knowledge: 
Students typfcally have much more knowledge than they use well. If students could somehow 
activate their inert knowledge and perform intemal cross-checks of gartjied knowledge, they 
might perform substantially better. 

Elementary PrnhlPm-qQlvinn .qtratP^iPS 

By asking themselves elementary "problem management" questions like "What am I trying to 
do now," "do I know a command that could help," "exactly what does the line of code I just wrote 
do if I hand execute it," and so on, students might make better use of their fragile knowledge 
base. Clinical experiments reported in Peridns, Hancock, Hobbs, Martin, and Simmons (1986) 
and Perkins, Martin, and Farady (1986) suggest that this is so. Unfortunately, students do not 
appear to probe with such questions as often as they might, a shortfall in elementary problem- 
solving strategies. 

Conflriftnrfl and r.nntf^| 

Finally, students often evince motivational problems that interfere with their controlling well 
their own problem-solving processes. For example, many students simply disengage from 
programming problems and commence a side activity or seek help as soon as the least difficulties 
emerge. These "stoppers" as we have called them are often quite capable of continuing on non- 
directive prompting, but do not seem to recognize their own abilities (Peri<ins, Hancock, Hobbs, 
Martin, & Simmons, 1986). 
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One way or another efforts to enhance instnjction in programming should address the triple 
problem of fragile knowledge, strategic shortfall, and confidence. Note that these difficulties as 
described here do not force wholesale reconsideration of what is taught in elementary 
programming instruction. After ail, the key commands, the fundamental operating procedures, 
and so on, surely need to be taught. Instead, the shOilfc identified could be taken to invite 
some sort of l^ooster shot" - some treatment to enhance .he learning set and strategic repertoire 
students bring to the enterprise of programming. 



The Notion of a Metacourse 

With this point in mind, we introduce the notion of a "metacourse." The goal of a metacourse 
is to provide mental models, problem-solving strategies, key concepts, and other structures that 
may help students to understand more deeply and wield mere artfully the knowledge they are 
acquiring during their regular instruction in a subject matter. In particular, the Metacourse in 
Programming discussed here offers students a mental model of the computer and how it wori<s at 
a level appropriate for understanding BASIC and its operation, a strategy for undeistar>ding new 
commands and relating them to the mental rmdel of the computer, several strategies for breaking 
programnrvng problems down into subproblems of various sorts, and other concepts and tactics 
designed to help students deal with the difficulties identified earlier (Peri^ins, Schwartz, & 
Simmons, in press). 

The metacourse is not a course - a complete remaking of the usual instruction. Rather it 
functions in a "meta" way, infusing a few important and often neglected conceptual elements into 
"business as usual." Thus, the Programming Metacourse is organized to allow the teacher to 
introduce key concepts periodically as the terni advances and students gain a knowledge base in 
BASIC. This infusion process, together with the metacognitive emphasis, are the factors that 
distinguish metacourse design from conventional curriculum redesign. 

Why might a metacourse offer an approach to enhancing instruction that evades some of the 
implementation difficulties outlined eariier? In comparison with a new course altogether, a 
metacourse is much more compact and its materials much less expensive. A metacourse does 
not displace, but merges smoothly with, existing materials and instructional practices. 
Accordingly, we suggest that a metacourse lends itself to wide-scale dissemination easily more 
than most efforts at curriculum reform. 

The results of a prelimina^ study reported in Peri<ins,Farady, Simmons and Villa (1986) 
indicated that the Metacourse was eminently teachable, with no major problems of teacher 
preparation. While there was some Duraging evidence that the Metacourse was having its 
intended effect, the data from this formative study were far from conclusive. 

In the foltow;ng pages, we report on two studies involving a number of treatment and control 
classroom which attempt to provide further evidence on ihe effectiveness of the Metacourse in 
settings where emphasis is placed on infusing the key concepts into the teacher's presentations 
throughout the entire course. 
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STUDY 1 

Method 

Setting 

The Programming Group's first large-scale study of the Metacourse was conducted within the 
context of another Educational Technology Center study, the "Laboratory Sites'* project. This 
broader study, described in detail in another report, aimed to involve the teachers using the 
Metacourse (as well as inte«veritions in two other subject matter areas) in the discussion and 
evaluation of the educational innovations they were implementing in their classrooms. 

Participation in the Laboratory Sites project meant that the teachers using the Metacourse 
received more support in their efforts to adapt the new materials to their own styles and 
classrooms than would normally be the case. The main extra supports were: monthly meetings 
with the research tearn to discuss issues pertaining to the implementation of the Metacourse; 
frequeni contact with a research assistant or the Laboratory Sites project liaison assigned to the 
Programming Group; access to all Educational Technotogy Center personnel, as weii as each 
other, via a personal computer communications netwcrt(, and a small stipend in compensation for 
their participation in collaborative research. 

Although not all of these supports may have been significant for all of the teachers in the 
experimental group, tal<en together they constitute an enriched support environment not 
normally present, and not present in our control sites. The control site teachers, however, were 
not faced with the problem of introducing and infusing new, innovative materials into their well 
practiced normal BASIC "urriculum. 

From the standpoint of the research, the participation of the Programming Group in the 
Laboratory Sites project created opportunities for very precise and timely feedbacl< from teachers 
about the effectiveness of the different lessons in the Metacourse. This feedbacl<, which would 
not othenA/ise have been available proved to be extremely valuable in guiding further revisions to 
the Metacourse. 

Subjects 

The experimental group consisted of 6 teachers of BASIC, «vho taught 9 classes and 132 
students at 5 high school laboratory sites. The control group pool consisted of 9 teachers who 
taught 13 classes and 239 students at 8 control sites. The large control group was used so that 
the groups could be matched if necessary on the basis of the general cognitive pre-test if overall 
pre-test scores indicated that one group or the other was significantly more able. The difficulties 
in data analysis encountered in the pilot study, in which the control group proved much more able 
on the general cognitive measures, might thus be avoided. 

All teachers were experienced programming instructors, beginning the semester with at least 
2 years of previous experience teaching BASIC classes. All of the classes were straightforward 
programming classes, meeting on average 5 times a weel< for forty-minute periods throughout 
one semester. 
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Procedures 

Teaching Intervention 

All of the experimental group instructors met with the Programming Group staff and research 
assistants early in the semester in order to go over the Metacourse materials and procedures. 
Each i6vncher was introduced to a particular research assistant who would observe two 
Metacourse lessons and one non-Metacourse lesson. 

As described in the introduction, the Metacourse is a series of instructional lessons 
designed to enhance and supplement the material typically covered in a first-semester BASIC 
course. One major concern is to provide students with a clear mental model of the computer in 
the BASIC environment and to help students internalize and employ the model when appropriate. 
Another goal of the Metacourse is to provide students with programming-specific skills and a 
conceptual framewori< that guide the student in initial understanding and subsequent application 
of material learned in class. 

The eight lessons are designed to be interspersed throughout the semester. Preliminary 
lessons provide students with a visual model of the computer and equip students with the 
conceptual framewori< for understanding commands, a framewort< that involves thinking of a 
statement in terms of its purpose, syntax, and action (semantics). Later lessons deal with 
strategies for writing good programs including breaking programs down into manageable chunks, 
thinking of code in temns of functional units called "patterns", and regular use of checking and 
debugging strategies. The components of the Metacourse are described in more detail below. 

The "naner romr^uter": A visual model of the computer . The concept of a mental nriodel is 
key in the design of the Metacourse. Recent wort< in the field of cognition underscores the 
importance of helping students construct robust models in various domains (cf. Beveridge & 
Paridns, 1987; Gentner & Stevens, 1983; Johnson- Laird, 1983; Mayer, 1976, 1981). Further, 
erDpirical wori< in the pedagogy of computer programming suggests that a stronger mental model 
of the computer can result in increased programming pertonmance (DuBoulay, 1986; DuBoulay, 
O'Shea, & Monk, 1981; Mayer, 1976, 1981,1985). 

In the Metacourse, students leam a visual model of what happens inside the computer 
during program execution to help them interpret exactly what the various BASIC commands do 
(See Appendix A). The rnodel is designed to help counter some of the problems of fragile 
knowledge and strategic shortfall we noted in students in our clinical wort<. Regular use of the 
model in learning commands and in envisioning the effects of single lines or chunks of code 
shouW serve to help build a robust knowledge base of command effects. In addition, the model 
can provide students with a fairly simple strategic tool for debugging. Using the visual model to 
imagine precisely the acttons of the computer at a trouble spot gives stoppers a tool to help thom 
get moving again and may assist haphazard movers by providing a focus for their activities. 
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Specifically, the model depicts variables and their values, characters on the screen, and flow 
of control. The model is functioiial rather than technical in nature in an effort to promote visibility 
and simplicity in the model (cf . DuBoulay, O'Shea, & Monk, 1981 ; Beveridge & Parkins, 1987). 
Thus, for example, the student thinks of a variable as the name of a box in memory with a number 
or character string in it. This model is called the "paper computer," because students receive 
paper forms displaying the visual model on which thry hand execute programs. In additton, an on- 
line version of the paper computer is being designed, a tutorial that presents the same visual 
model of the computer arKJ steps through a program line by line, making the requisite changes in 
the compi ^er state as each line is executed. 

PufpQse-svntax^actlon' A framework for r^ronram constituents. In the MetaCOurse Students 
also learn an analyttoal scheme for understanding comniands and command lines. The scheme 
encourages students to consider the command's purpose, its legal syntax, and its actton in the 
computer world as shown on the paper computer. The purpose-syntax-action framework is used 
to help students recognize, attend to, and organize the important features of a new command 
when it is first encountered. Additionally, the framewor1< serves as a problem-solving aid when 
students write programs. In considering how to accomplish a particular programming task 
students are asked to think carefully about what commands they know whose purposes ntay 
accomplish the task at hand. When writing command lines, students are encouraged to attend to 
the syntax, and to envision the actton of the command, that is, the effect of the Ime on the 
computer. Students are also encouraged to use a "minimanual", a qutok reference guide that 
includes the key BASIC commands organized according to the purpose, syntax, and action 
questions, with examples (See Appendix B). As students gain programming experience and 
become comfortable in using the framewori< for individual commands, they are introduced to the 
utility of emptoying a similar organizational strategy for larger "pattems" of code. (Patterns will be 
discussed below.) 

The chotoe of terms in the framework is derived in part from the contrast between 
programming pragmatics, syntax, and semantics, and in part from a model for learning developed 
by Peridns (Peri<ins, 1936a,b). Purpose has a major role in the triad as a means to counter some of 
the problems of fragile knowledge, specifically that of inert knowledge. In eariier clinical studies it 
was found that often students had a knowledge of relevant command structures, but could not 
retrieve them, apparently failing to make the connection between what needed to happen in the 
program and the commands that would serve the purpose (Pertdns & Martin, 1986; Pert<ins, 
Martin & Farady, 1986). The attentton to syntax is based on eariier observations noting the degree 
to whk:h BASIC and LOGO program perfomnanco among novice programmers can be affected by 
problems of a purely syntactic nature (Pertdns, Farady, Hancock, Hobbs, Simmons, Tuck, & Villa, 
1986; Peri<ins, Farady, & Martin, 1986). The emphasis on the action or semantics of a command 
line stems from the desire to help students construct and utilize a robust knowledge base of 
command effects as described above in the discussion of visual models. 
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Patterns: thinklnip beyond individi^al commands. The Metaoourse emphasizes the 
importance of organizing programming knowledge not only in temis of individual comrnands but 
also in ternis of multiple Ones of code that work together to accomplish a parttoular task or subtask. 
Such recurrent schema are called "patterns " a temi roughly synonomous with the '^programming 
plans" de$crit)ed by Soloway and colleagues (Soloway & Ehrlteh, 1984; Joni & Soluway, 1986). 
Patterns pruvkle an intenTtediate level of analys.3 between the whole program and individual 
command lines, and offer a way of helping students to organize and comprehend code used for 
frequent pro^^ramming tasks such as counter variables, certain compound oondittonal branches, 
and so forth (See /ippendix C). The Metacoui^e stresses the importance of patterns as an 
intemnediate level coding strategy as well as a debugging strategy. 

Writing a program: Planning, coding, checking, and debugg in g . Test results and teacher 
conments from our previous study irKJicated that students needed additional instruction in 
planning and writing whole programs. Thus the revised Metaoourse provides students with a 
procedure for writing programs that emphasizes the importance of planning, checking, and 
debugging in additton to the actual coding of a program. Another major point of emphasis is that 
writing programs is a process of refinement, involving multiple rounds of these four activities. 

As a first step in top-level planning, students are encouraged to think in terms of 
"interactions" with the user. The interaction refers to the sequence of computer outputs and user 
inputs that occurs as a user uses a program. This heuristic offers a concrete beginning point for 
the novice programmer, wtio frequently experiences difficulties in moving from a given proulem 
statement to the initial stages of the taskdecorrpositton. For many programs, an outline 
according to the "rounds" of interactk)n provides a deoomposHton into subunits that amount to 
subproblems in the programming task. An initial focus on interactions as the student begins to 
program can help to avoid the frequent problem of moving from a problem statement directly to a 
coding phase without sufficient thought and effort devoted to planning. 

From this initial breakdown of the problem, students are asked to conceptualize program 
sectk)ns in terms of the patterns that might serve the section purposes. The patterns provide the 
student with the tools to manage and create code above the level of a single command line. 
Students are taught the utility of having a repertoire of patterns that are portable across programs. 
Thus, for example, if a student recognizes the need in a program to implement code that will trap 
unreasonable inputs from the user, the student can call on a iDullcn^roofing" pattern to help with 
that task. This simple pattem includes a condittonal branch that rejects inappropriate inputs, 
prompting the use for another input. 

At each level of the planning and coding stages students are encouraged to consider the 
purpose of chunks of code and the cctk)' those chunks actually effect. Students are taught to 
check their code, mentally simulating the action of the program line by line to catch the '*easy" 
bugs before the program is actually run and tested. The Metaoourse also emphasizes that bugs 
and debugging are an inevitable and integral aspect of program production, not a reflection of 
poor programming. 
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Afifi^ssment Pmcflrinrftg 

A pretest in general cognitive skills was administered at the beginning of the terni. At the 
end of the term, the cognitive skills test was readministered to test for possible transfer effects, 
and a general BASIC skills test was given to test for mastery of BASIC. Both tests are described 
below. 

CognltlvQ skills nrfltftst-posttest rAnnfltvtiy n\. A cognitive skilis test was developed with two 
purposes in mind. First, such a measure might be expected to correlate with programming 
performance (most cognitive skills are interrelated), and thus sen^e as an indk:ator of level of 
general student ability in comparing treatment and control students. In addrtton, it seemed 
sensible to probe the possibility of transfer of cognitive skills. In general, f irxlings on transfer from 
programming have been negative (Kuiiand, Pea, Clement, & Mawby, 1986; Mawby, in press; Pea 
& Kurland, 1984a,b: Sotomon & Perkins, in press), nor was the Metacourse specifically designed 
to pronfiote transfer. However, there have been occasional positive findings (Clements, 1985; 
Clements & Gulto, 1 984) and the issue has great currency, wan-anting an effort to examine the 
question of tra^-^fer in the present study. 

The cognitive skills instrument was designed to test skill in formal syltog'stto reasoning, 
complex linear reasoning (e.g. ""If the day after tomonow were Thursday, what woukl the day 
before yesterday beT), field-independence and planning (a task of counting the number of 
triangles in a complex diagram). In additton, the test incorporated an algebraic version of the well- 
known students-and-professors problem (Clement, Lochhead, & Monk, 1981) where students 
typically have great diffkulty translating a simple algebra word problem into an algebraic equation. 
Soloway and his colleagues have suggested that computer programming experience may help 
students deal with this sort of problem more effectively (Ehrlteh, Sotoway, & Abbot, 1982; 
Sotoway, Lochhead, & Clement. 1982). A further item relating to accuracy and precision of 
obsen^atton and description, required the student to accurately describe a complex geometric 
figure in order for another student to be able to recognize it amongst a set of similar complex 
geometrto figures. 

The cognitive skills test also included a problem of a type quite similar in character to a 
programming problem involving combining patterns in a program-like way. Students had to 
generate a descriptton of some events using a restricted language containing the words "repeat" 
and "deckle". (The design of these problems was suggested by Ellen Mandinach of the 
Educattonal Testing Sen^lce); in prior research, when the rare case M transfer from programming 
has been found, it has emerged most often on tasks with marked similarity to programming (cf. 
Kuriand, Pea, Clement, & Mawby, 1986; Linn, 1985). 
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Finally, in addition to the purely "cognitive'' aspects of the pretest, we also included a series 
of questions designed to address a particular attitudinal factor, locus of control (Rotter, 1966). 
Research in this area (Dwecic & Bempechat, 1983; Dweck & Ucht, 1980), zs well as studies of 
peiformance in the programming domain (Perkins, Martin & Farady, 1986; Zelman, 1985), point to 
the importance of attending to feelings of confidence and control in student perfonnance. Thus 
we included a 10 question subtest derived from the "attribution of intellectual responsibility scale" 
(Rotter, 1966), measuring intemality/extemality of control in positive and negative intellectual 
situattons. 

BASIC test ^Appendix ). The BASIC test was a fairly starxlard paper and pencil 
programming test, comprising 16 items. The problems were fonriulated to evaluate certain 
general programing skills - the ability to hand-execute, debug, break a problem into subtasks - as 
well as to test knowledge about programming commands typically presented in an introductorv 
BASIC course (e.g. PRINT, LET, INPUT, FOR/NEXT, IF-THEN). The test was also designed to 
include certain language independent bugs identified by Pea (Pea, 1986) . 

The first nine problems tested simple production and recognition skills, usually requiring 
production of only one line of BASIC code using one command statement. The last seven 
problems included three hand-execution problems, one debugging problem and three longer 
programming problems requiring somewhat more complex structures such setting up counters, 
use of for-next loops and the like. ITie 16 problems are briefly described below. 



Simple one Una command pmhiems (1 . 9): 

1 . PRINT-string: The student is asked to write the code that would result in a specific 
string 

being printed on the screen. 

2. PRINT-number: A specific number is to be printed on the screen. 

3. INPUT-number A user is prompted to type in a number. 

4. LET-number Set a variable to a number. 

5. LET-expression: SO a variable to the result of an operation. 

6. LET-string: Set a variable to a string. 

7. IF/THEN: If a given condition holds, print an expression. 

8. FOR-k>op: fill in blanks to complete a simple For/next k)op. 

9. GOTO: Fill in a blank to complete a Goto statement. 

Hand ftyftnution problems (10. 11 and 13): 

10. Given a short program, show what will appear on the screen when the program is 
executed. 

1 1 . Garden path : Same as 1 0 above, except that what will appear on the screen when 
the program is executed is not what one would expect - the hand-executor is led 
down a garden path. 

V 13. Parallelism bug: Same as 1 0 above, except that the student is given a program in 

which the indivMual commands are conBCt, but are presented in the wrong order. 
This was designed to test for skill in recognizing v/hat Pea (1985) calls a parallelism 
bug, the assumption by programmers that the computer is able to execute more than 
one line of code simultaneously (in which case order would not be important). 
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Dshtigg in Q probtem (12 - see also problerrs 1 1 and 1 3): 

12. Egocentrism bug: The student is given a program with an INPUT statement missing 
because they often assume that the computer is aware of vhat they are thinking 

14. FOR-loop: Using a FOR/NEXT loop, write a program that will print the numbers 1 
through 10 on the screen. 

15. iP-loop: Same as problem 14, except that the loop is created using an IF statement. 

16. Complex: The student is asked to write a more complex program tnat requires an I 
INPUT statement Iriside a FOR k)op tfiat accumulates a sum. 

Classroom Qhsftrvatinns 

Researchers conducted classroom observations at all treatment sites, systematically 
xecoT^&Xin a number of features of the classroom dynamics and instructional style, as well as items 
related to the Metacourse treatment itself. Observattons included two Metacourse lessons, as well 
as one regul?*^ class, in each case conducted by a single research assistant assigned to observe 
^ ne or two teachers. Assignments of multiple obsen^ers to a class were not made for three 
reasons. Rrst, we wished to be as non-obstrusK/e as possible and thus each teacher and class 
coukJ get comfortable with a single outside observer. Further, it woukJ have exceeded our 
personnel resources to provide mutiple observers. Finally, since our main comparsions were 
between Metacourse and non-Metacourse lessons in the same classes, inter-rater reliability was 
not an issue. 

The majority of the items on the observation instrument (See Appendix F) were rated on 7- 
point Likert scales (where 7 Is the highest rating and most desirable score). These included 
situatk)nal factors such as student- teacher and student-student interaction, student 
responsiveness and engagement, and teacher presentattonal style. Items relating specifically to 
the Metacourse -ncluded fidelity to the Metacourse and infusion of Metacourse concepts into the 
Metacourse lessons. In the area of classroom interacttons, the scales measured amount and 
quality of interc^^tons between students and teachers and among students. General types of 
student particle v '^on were scored, measuring the degree to which students engaged in process- 
a'^d produc*<T^i^r participatton. Students (as a group) were also rated on preparedness for 
the mateiK/ ps^ds o^.ed. Teachers were also rated on mode of instruction (e.g., presentattonal, 
interactive} on Jr/^w ox comfort with the material presented. Obsen/ers noted whether the 
instruction ; \^.\u<:?.> ^{Vj . edging activities betweei: programming and other acadeniic disciplines 
or real-life r'^ua:^ ^is. Rnally, observers used the 7-point scale to give an overall rating of 
effectiveness of instruction. 

In the oDsen/ations of Metacourse lessons, fidelity to the lessons was rated, based on 
adherence to various aspects of the lesson (introduction, lecture, participation, explanation, and 
exercises). The Metacourse lessons were also rated according to the degree to which the basic 
principlos of the Metacourse had been integrated into the lesson. 



Programming 11 



Student QiiftRtionaire 

At ttie beginning of ttie semester, students filled out a short questionnaire designed to help 
assess their previous experience with computers and computer programming (Set» Appendix G). 
Items were designed to determine the students' general experience with computers in the 
schools, with computers in the home, and in other outside activities (e.g., computer camps). In 
addition, students were asl<ed to report previous experience in the BASIC programming language 
as well as other programming languages. 

Homework 

Most of the nine Metacourse lessons included worksheet problems for students to complete 
in class and homework assignments to be completed outside of class between lessons. The task 
of sending homework and worksheet papers to ETC was rotated anong the teachers so that, for 
each lesson, different teachers were asked to retum their students' papers. These were xeroxed 
and returned as soon as possible to the teachers. Teachers returned 83% of the sets of student 
homework and worksheets asked for (25 out of 30). Teacher response was better toward the end 
of the Semester than the beginning. Each student's homework was '"graded" and the total 
nurDber of students making a particular number of errors on each problem was recorded. 

Coding Profiftriures 

CognitlvQ prfttftst^posttest . The cognitive test included several types of problems. The- 
days-of-the-week problems and the students*and* professors (formula) problem were scored 
either 0 for correct, or 1 for incorrect. In the triangle-count problem, the correct answer was scored 
0. If the student reported more than the correct number of triangles, the score was the excess 
with a minus sign; if fewer, the score was the shortfall with a plus sign. 

The description-of-geonf)etric-f]gure problem and the repeat s-and-decides problems 
required special conventions because of the complexity of possible responses. In the geometric 
problem, the following categories were scored: 1) right figure kept, 2) geometric shape omission, 
and 3) positton omission. If the correct figure was kepi the student scored a 0, and one point was 
added for each shape in excess kept. For the shape category, the student scored a 0 for coi rect 
descriptton of shapes, 1 if one shape was omitted, 2 if both shapes (square and rectangle) were 
omitted. If the correct positioning of the squares was conect the student scored a 0, 1 for one of 
the relationships was missing, 2 if both relattonships (top-bottom/spread out) were omitted. 

In the repeats-and-decides problem, fbrm, order and content were defined as scoring 
categories. Form referred to the writing of commands in the correct format. Order referred to the 
correct ordering in a sequence of commands. Content concemed whether the student's repeats 
and deckles "program" wouM do the assigned task. All categories were scored 99 if the student 
dki not try the problem. Othen^^ise the categories were scored on a 0 to 4, where 0 designated a 
perfect answer, 1 , a single error and so forth up the scale to 4 which indtoated insufficient data/ 
misunderstood task or for the form category, repeats and decides not used. 
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The Attribution of Intellectual Responsibility scale was scored in the usual manner in terms of 
which altematlve (internal or external locus of control) was chosen on the 5 positive and 5 
negative intellectual situations. 

BASIC test . The Students* perfomnance on the BASIC test was scored in a number of 
different categories. Manyof the categories suited only certain problems, for instance, the 
counter category applied only to problems that included a counter. Each of the 16 problems was 
scored for errors by noting whether or rx)t a student's response Included certain specific features. 
Responses to Problem 1 (PRINT-string), for example, were scored for two features, or error 
categories. To be enfor free, a response needed to include a correct PRINT command (the first 
error category) as well as a quoted string (the second category). The sample response (1) below 
would be scored as having 0 errors - it is correct. Response (2), however, would be scored as 
having 1 enror; the PRINT is correct, but the quote category is scored as incorrect because the first 
quotation mari< is missing. 

(1) PRINT "the answer is catl 

(2) PRINT the answer Is catl 

Some problems called for an "extraneous statements" category. This was scored when an extra 
statement or set of statements was added to the program, which may or may not have resulted in 
an error, but showed clearly that the student had an improper model of machine semantics. 
Several other categories corresponded to the use of various program constructs; for instance the 
positioning of PRINT statements, FOR-NEXT statements and the lil<e. The number of error 
categories scored for a problem ranged from one in problem 9 (the fill-in-the-blanl( GOTO 
problem) to ten in problem 16 (the complex production problem). 

Scoring pmredure. Using the scoring system described eariier, 4 scorers on the cognitive 
test and 2 scorers on the B ^SIC test woriced independently to code the students' responses to 
the test, crosschecking periodically with one another and establishing policies to clarify the 
scoring system. The coding was blind with respect to whether the response came from the 
treatment or control group, or from a pre or post-test. While every student response was not 
coded by all scorers, the scorers all coded and discussed a random subsample. Policies that 
could affect prior scoring were applied retroactively. 

After about ten percent of the data had been scored the scorers deemed themselves to be 
adequately calibrated. The remainder of the coding provided the basis for calculating interjudge 
agreement. Disagreements were discussed and resolved, though original scores were retained 
for calculating the agreement. All the coding was used for the data analysis in other respects, on 
the grounds that divergences had been discussed and resolved and principles applied 
retroactively to the coding before the calibration. These procedures resulted in very high 
interjudge reliabilities on both tests with correlations of .90 or greater in all cases. 



ERIC 



Programming 13 



In order to determine whether any of the judges tended to score significantly higher or lower 
than the others, the mean ^re and standard deviation for each judge were calculated and 
compared for the totals on both the BASIC and Cognitive tests. In each case ANOVA indicated 
no significant differences among the judges. Therefore, for those tests that were multiply scored, 
the judges scores were averaged, while for those tests that were scored by only one judge, that 
score was used in all analyses. 



Results 

Fidelity to Metacourse Lessons as Written 

Of critical importance in evaluating any educational intervention are questions related to how 
it is actually implemented. We observed a number of Metacourse, and non-metacourse lessons 
taught by each teacher in order to assess how the teachers and students responded to the 
materials. We found that the teachers covered the content of the Metacourse with good fidelity, 
usually rated around 5, where 7 was described as teaching the material "almost exactly as written" 
and 4 indicated "the same or very similar content of lesson as written, but adapted and 
paraphrased". This was consistent with the rating observed in our preliminary study, also about 5. 
It appears that the lessons were either paraphrased or presented in a manner very similar to how 
they were originally written. 

Teachers were usually rated as covering three quarters of the material in each lesson quite 
adequately and also as referring appropriately during the Metacourse lessons to the major 
principles stressed in the Metacourse, ie. the paper computer, purpose, syntax, action, patterns, 
with the exception of the minimanual which was very rarely mentioned. Thus the minimanual may 
be an underutilized resource, at least in terms of its integration into the instructors' presentations. 

In contrast to the generally good coverage of the lessons and principles of the Metacourse, 
teachers had much more difficulty in pointing out bridges or applications either to other academic 
disciplines or problems in real life. Such bridges were noted in only about 10% of the classes, 
whether they were Metacourse or non-Metacourse lessons. 



Homeworic 

The primary purpose for the homework/worksheet analysis was to determine whether or not 
teachers and students were actually able to use the assignments. The analysis indicated that they 
were. Students seemed to find the assignments appropriate and of reasonable levels of difficulty. 



The secondary purpose was to determine where the Metacourse lessons might be 
improved. In general, the homework analysis was most useful in detecting problems, not 
strengths, in the Metacourse and was best interpreted in conjunction with teacher comments. 
This analysis, atong with teacher comments was used in preparing notes to be used for teachers 
in a future study in which potential problems in implementing the Metacourse could be 
addressed. 

Some specific problems noted during the analysis included the following: \6) a number of 
students had difficulty hand executing simple programs that included PRINT and LET statements 
containing expressions; (b) many students did not adopt the pointer that the Metacourse 
suggested they use to keep track of the cun-ent line during hand execution; (c) PRINT/INPUT 
combinations confused many students - the PRINT statement preceding the INPUT statement 
often interiered with hand execution of the INPUT statement; (d) the Assume User Types area of 
the computer worid was not used by some students, perhaps because this was not explicitly 
modeled by teachers; (d) many students did not attempt to Mentify patterns in problems, 
presumably because they did not understand this concept; (e) a common hand execution 
problem with somewhat more complicated programs was to miss one of the many required steps 
(e.g., changing a variable in the Variables Area of the Computer Worid each time it is encountered 
in a FOR/NEXT loop). Many of these same difficulties manifested themselves later on in the 
BASIC end of semester test. 

Diffarences between Metacourse and Ordinary Lessons 

As in our prevtous study, obsen/ations from the experimental sites were examined for 
differences in student and teacher behavior when Metacourse lessons were taught as compared 
to when ordinary lessons were taught. Of particular interest were any major differences in what 
went on in dass during such lessons and whether these seemed disruptive or benefk:ial. Since in 
each case the obsen^ations of a paficuiar class were made by a single observer who was not blind 
to the type of class being conducted, thbse results must be interpreted with cautton. However, it 
shoukJ be noted that the obsen/ers were generally quite experienced, the variations reported 
between teachers were not large, and the general pattern of results appears quite reasonable. 

There were few surprises in the results we obtained. As in our previous study the teachers 
seemed somewhat more comfortable when presenting their own lessons (mean rating of 6.9) 
compared to the new Metacourse material (mean rating of 5.8). However, both scores are qu'rte 
high (7 point scale) indicating general comfort with both old or new material. Similarty were found 
large differences between teachers' styles with non-Metacourse lessons rated as almost twice as 
Interactive (60% vs. "2%), while somewhat less presentattonal (26% vs. 15%) than Metacourse 
lessons. During Metacourse lessons the data suggest that more time was spent in lecture (25% 
vs. 12%), demonstrations (20% vs. 11%), and in hand executfon exercises (21% vs. 9%), and 
less time in discusston (23% vs. 40%), and "other^ exercises (18% vs. 48%). Student behavior 
during both types of classes was rated as quite responsive, attentive, engaged, and interactive 
with their teachers (ratings generally over £ in both types of classes; however, there was 
apparently more interactton anrwng students during the non-Metacourse lessons (mean rating of 
5.4 vs. 3.4). No differences were observed in temis of the nature of such interactions, with both 
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teachers' and students' interactions described as process-oriented and product-oriented about 
half the time, with very little disciplinary interaction occurring at any time. Students seemed quite 
prepared for either type of lesson (5.1 for the Metacourse, and 5.7 for non-Metacourse lessons). 
Finally, despite the differences that were noted, the overall effectiveness of each type of lesson 
was judged quite high (5.4 for Metacourse, and 6.0 for non-Metacourse lessons). 

Group Differences 

The findings above seem to indicate that despite differences between normal and 
metacourse lessons, the Metacourse materials were adequately covered although both the 
observations and teacher self reports indicate considerable difficulty in making connections 
between the material and other academic and life problems. 

However, assessing the impact of the Metacourse also requires equivalence of the treatment 
and control groups on other factors that might influence programming achievements, or statistical 
corrections as needed. Consequently the data were examined by group to check for significant 
differences extant before the treatment. 

Previous Pynftriance with Computers and Computer Programming 

Most of the students in both groups had little prevtous experience with programming 
languages. Three quarters of the experimental group and 84% of the control group had no 
prevtous exposure to BASIC. Those few with experience, however, had minimal exposure 
(typically a week or two here or there). While 24% of the experimental group had some 
experience with another programming language compared to only 5% in the control group 
(p<.05), this consisted in all but 2 cases in each group of a brief exposure to LOGO, some years 
back. Finally a little less than half our subjects in each group (44% in the experimental and 49% In 
the controls) had a computer in their home. Thus with respect to these types of experiences the 
groups seemed not substantially different. 

Cognitive Pre-Test 

The cognitive test was assumed to be an indicator of general ability that might relate to later 
BASIC perfomnance, and in fact significant positive correlations were obtained In both groups ( 
r».69, p<.01) in the experimental group and (r«.40, p<.01), in the control ^jroup. The Analysis of 
Variance indtoated no significant difference in homogeneKy of slopes between the two groups 
(F- 1.345, n.s.). 

Overall the control group performed signifk:antly better than the experimental group (p<.01), 
commiting on average about 12 errors compared to 16 errors for the experimental group. The 
range was from about 6 to 29 enfors in the 9 experimental classes compared to 5 to 1 6 errors in the 
13 control classes (A more detailed analysis of subtests is presented later when the issue of 
transfer is considered.) 
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Thus the control group appears to be of somewhat higher ability than the experimental 
S'oup, yet as the data will indicate the experimental group performed significantly better on the 
BASIC test administered at the end of the course. 



Impact of Instruction on Mastery of BASIC 

Tablel (next page) indicates the performance of experimental and control groups in terms of 
mean errors on each of the 16 BASIC Test problems as well as subtotals for four types of 
problems and the total test. Despite the apparent higher general ability level in the control group 
the experimental group averaged about 5.5 fewer errors, nearty half a standard deviation overall, 
than the control group. They produced about 77% correct responses compared to about 66% for 
the controls (a maximum of 61 errors was possible). 

Furthernx)re, the experimental groups did significantly better than the controls on all major 
categories of problems, with the smallest advantage coming on the simple one line command 
problems and the production problems (about a third of a standard deviation), and the largest on 
the hand execution problems (nearly two thirds of a standard deviation). 

Results for individual problems within three of the major categories - Hand execution. 
Debugging and Production - show that the experimental group had significantly fewer errors on 
every problem except one requiring the use of an IF/THEN statement to create a loop 
(Problem15). In fact, both groups found it more difficult to write a program using IF/THEN after 
being asked to create a program that generated identical results but used the more natural FOR 
loop (Problem 14). 

Interestingly, there was also no significant difference between the two groups on the 
IF/THEN problem within the other major category - Simple one line commands. The only 
problems within this category for which there were significant differences included one which 
asked students to set a variable to a string (Problem 6), one requiring the completton of a 
FOR/NEXT loop (Problem 8) and one in which students needed to recognize when a GOTO 
statement was needed (Problem 9). These last two problems (8 and 9) differed from the other 
problems within this category because partial code was provkied that, presumably, needed to be 
read and understood before being completed. In this respect, perhaps, these problems were 
similar to the Hand-executton problems. 

The BASIC test problems described above often contained the same features or error 
categories. This aOowed us to undertake a finer analysis of how students handled a particular 
feature across and within various problems. These additional error categories are described 
below, and the results presented in Table 2. 
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Table 1. Errors (Mean Number) on Each Problem of the BASIC Test 



Simple One Line rnmmand Prnhlems 





1 


2 


3 


4 


5 


6 


7 


8 


9 


Total 


Exp. 


.13 


.13 


.53 


.11 


.25 


.71 


.88 


.66 


.?7 


3.67 


Cont. 


.08 


.11 


.72 


.16 


.22 


1.14 


.88 


1.13 


.49 


4.92 


Diff. 


.05 


.02 


-.19 


-.05 


.03 


-.43"* 


-.00 


.47*** 


-.22*** 


1.25"* 



D/S.D 



Hand Execution Prohlems DeBuanino Prnhtem 





10 


11 


13 


Total 


12 


Exp. 


.28 


.74 


1.65 


2.67 


.61 


Cont. 


.73 


1.47 


1.94 


4.14 


1.05 


Diff. 


-.45"* 


-.73*** 


-.29* 


-1.47*** 


-.44 


D/S.D. 






.61 




.48 



Production PrnhlRms All Problems 





14 


15 


16 


Total 


Total 


Exp. 


1.13 


2.83 


3.41 


7.36 


14.31 


Cont. 


1.60 


3.11 


5.07 


9.77 


19.88 


Diff. 


-.47* 


-.28 


-1.56* 


"2.41* 


-5.57*" 


D/S.D. 








.34 


.45 



Experimental Group (n-120) 
Control Group (n-224) 



* p<.05 
** p<.01 
*" p<.001 
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Simple one line command error rfltegories (Problems 1-9): 

1 . PRINT (Problems 1 -2) : The word PRINT is not present or is spelled incorrectly. 

2. Quote (Problems 1 and 3): The string following the PRINT command is incon'ect or is 
not preceded by a quotation marie 

3. LET (Problems 4 - 6): The word LET is not present or is spelled incorreuiy. 

4. Assignment (Problems 4 • 6): The assignment statenront following the LET command 
is incorrect. 

5. ACTION (Problems 7 - 8): The expression following THEN in problem 7 is incorrect; 
the expression between the FOR and NEXT commands in problem 8 is inconrect. 

6. GOTO (Problem 9): The GOTO command or the line number following it is incorrect. 

7. INPUT (Problem 3): The word INPUT is not present or incorrect. 

8. MISC. (Problems 2,3,7,8): Various errors occuring in only one problem. 

Hand-execution c^rrQrnatftnnriAg (Problems 10, 11 and 13): 

9. Prompt/Input (Problenis 10, 11 and 13): The output generated from INPUT 
statements does not include the correct prompt and/or input. 

10. Output (Problems 10, 11, and 13): The output generated from PRINT statements is 
incorrect. 

1 1 . Minor execution: Any execution errors not covered by other categories. 

12. Fomiat: Output is not property fomiatted. Debugging enx)r categories (Problems 11, 
12, 13): 

13. Bug-11 (Problem 11): The connect ("unexpected") output is not generated. This 
category does not include categories lil<e minor execution, output and so on that 
were also scored in Problem 1 1 . The only feature considered is the one directly 
related to debugging. 

14. Bug-12 (Problem 12): The correct INPUT statement is not added to the program (as 
above, other "non-bug" categories are excluded from this category). 

15. Bug-13 (Problem 13): The correct ("unexpected") output is not generated (as in 
category 13, other "non-bug" categories are excluded). 

Productten pn-or cate^orifts (Problems 14, 15 and 16) 

16. Counter(Problems14and15): The counter following the FOR com-^<*. 
incorrect. 

17. Actton (Problems 14 and 15): The major action (PRINT X) is incorrect. 

18. FOR (Problems 14 and 16): The FOR statement is missing or incorrect. 

19. NEXT (Problems 14 and 16): The NEXT statement (NEXT X) is incorrect or missing. 

20. Variables (Problems 14 and 1 6): Incorrect number or type of variables used. 

21. Extraneous (Problems 14, 15 and 16): The program includes unneoded commands 
that interfere with the running of the program or that wdke the solution clumsy. 

22. Minor syntax (Problems 14, 15 and 16): This category includes erro-'S not covered 
by other categories that ocur in more than one problem. 

23. Misc. (Problems 1 5, 1 6) : Various error types appearing in only one problem. 

Table 2 (next page) presents results from these individual enror categories within and across 
problems. Like the results presented in Table 1 , these results indicate that the Metacourse 
group made significantly fewer errors in each of the four major categories. However, they also 
Indicate that this did not occur uniformly within each major category. That is, the experimental 
group, although never commiting significantly more errors than the control group, did not perform 
significantly better than the controls on all error categories. 



ERLC 



Programming 19 



TABLE 2. Types of Errors (Mean Number) on the BASIC Test 



Simple One Line Command Errors ^Prnhiems 





1 


2 


3 4 5 6 


7 


8 


9 




PRINT 


QUOTE 


LET ASSIGN ACTION GOTO 


INPUT 


Misc. 


TOTAL 


Exp. 


.04 


.23 


.45 .64 .70 .27 


.21 


1.14 


3.67 


Cont. 


.05 


.22 


.80 .72 .93 .49 


.35 


1.36 


4.92 


Diff. 


-.01 


.01 


-.35"* -.08 -.23** -.22*" 


-.14 " 


-.22 


1.25" 



Hand FYPriition Frrnrs f Prohlfims 10.11 1.?^ 





9 


10 


11 


12 


13 




PROMPT-INPUT 


OUTPUT 


MINOR EX. 


FORMAT 


TOTAL 


Exp. 


.70 


.13 


.53 


.04 


1.40 


Cont. 


1.87 


.08 


.60 


.06 


2.60 


Diff. 


-1.17*" 


.05 


-.07 


-.02 


-1 .20" 



nebuQgina Prmrs fPrnhlft ms 11.12.13^ 





13 


14 


15 






BUCa-11 


BUQ-12 


BUQ-13 


TOTAL 


Exp. 


.48 


.26 


.78 


1.53 


Cont. 


.76 


.48 


.78 


2.02 


Diff. 


-.28*" 


-.221*" 


.00 


-.49*" 



Production Prmrs ^Probifims 14.1.S.16^ 

16 17 18 19 20 21 22 23 
COUNT. ACTION £QB NEXT YAH- EXTRA . MIN SYN . MISC. TOTAL 



Exp. 


.48 


.42 


.32 


.34 


.52 


1.02 


. 96 


3.30 


7.36 


Cont. 


.58 


.57 


.64 


.76 


.74 


1.35 


1.16 


3.98 


9.77 


Diff. 


-.10 


-.15 


-.32*" 


-.42*" 


-.22** 


-.33** 


• .10 


-.68 


-2.41** 



Experimental Group (n«1 20) ♦ p<.05 

Control Group (n«224) p<.01 

*** p<.001 
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With respect to the "simple one line command errors" (in the first 9 problems) the 
experimental group did no better than the controls on a number of simple error categories such as 
PRINT and quote and assigning a value that a variable should take; however, they made about half 
the enors of the control group when producing simple LET statements (enor category 3) and 
about one quarter less errors than the controls in specifying the action of simple IF/THEN 
statement or '-OR loop (category 5). 

As noted above, the experimemal groups made significantly fewer hand execuiiun enors on 
problems that required creating a screen display when given a program (Problems 10, 1 1 and 13). 
It appears that the experimental groups' superiority is due to its ability to represent what a prompt 
and input statement iook like on the screen (Error category 9), since there were no differences in 
the output , minor execution and format error categories (1 0,1 1 ,&12). 

if we in turn ignore the prevtous error types that may occur during hand execution and took 
specifically at the aspects of execution that are directly related to program bugs, we find that the 
experimental group is signiftoantly better detecting unexpected output (Error category 13 from 
the Garden Path problem). They are no Aer, however, at detecting the parallelism bug 
(Category 15). Bug-12 is periiaps a more realistic debugging problem. The students were given a 
program and simply asked to correct as well as find a bug. In this situatfon, the exper'-nental group 
was again signif teantly better than the non-metacourse group. 

The final set of error categories to be consklered are those related to the production of 
whole programs. When given three programming problems, the group with Metacourse training 
handled FOR/NEXT toops better, was better able to choose the conrect number and type of 
variables to use and produced signiftoantly fewer unnecessary statements (error categories 1 8 
through 21). There were no signflcant differences between the two groups on the remaining 
productfon error types. As was the case with all other variables analyzed in this study, the control 
group dkl not make significantly fewer errors in any of the categories investigated. 

Transfer 

As noted in our prevtous reports, while the cognitive posttest provided an occasion to 
examine possible transfer of cognitive skills from programming instructton it shouW be recalled 
that such a finding was not necessarily expected from the Metacourse treatment. Most research 
on the impact of programming instmctton on cognitive skills has not found substantial gains, 
except under very special condittons (ie- speciffc emphasis on transfer throughout a program, 
one-or-one tutoring, and tests composed of problems formally similar to those encountered in 
the programming course). In contrast, the Metacourse was designed speciftoally to advance 
programming skills rather than to promote general transfer. Nonetheless, the data presented an 
opportunity to explore the issue within this limited context. 
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Impad on General Cognitivfl Shills 

As indicated previously the general cognitive tests were composed of 6 separate subtests 
(See Appendix D). There was almost no variability of the Attribution of Intellectual Responsibility 
subtest. Both experimental and control groups attributed an average of between 7.7 and 7.9 
questions (out of a possible 10) to internal factors, both on pre and posttests. Analysis by positive 
or negative outcomes yielded similar results. Thus most of our sample attributed their successes 
and their failures primarily to internal causes, both prior to and after the BASIC programming 
course. This variable was inerefore excluded from all further analyses. 

Table 3 (next page) presents the performance of each group on each of the other 5 subtests 
and their total score for both pre and posttests. 

As indicated previously the control group made significantly fewer emors on the pretest 
overall than the experimental group (p<.001). Analysis by subtests indicated that this superiority 
manifested itself on three of the five subtests (conditional reasoning, precise visual description, 
and repeats and decides), while no initial diffe^'ences were found on the visual planning or 
professors/students problems. 

Surprisingly the experimental group improved by about three fewer errors on the posttest, 
compared to virtually no gain for the centre, group. It is unlikely that this highly significant outcome 
(p<.001 ) would be due merely to regression to the mean. Analysis of the data from the 
connponent tests indicates that this result is due almost entirely to the considerable improvement 
of the experimental group or !he Repeats and Decides subtest, compared to a modest decline in 
performance on the part of the control group on that same problem. Thus some evidence of 
transfer was observed, but It occurred only on the problem most closely related in its formal 
stmctureto that of producing coherent comands in a programming language. 

Teachers' Reactions to the Metaoourse 

Since this sT was conducted within the laboratory sites context described eariier, we 
received f requeru .^^ jbacic from the teachers concerning the metacourse materials. The 
teachers were in general quite positive in their evaluation of the revised metacourse. The utility of 
the visual model of the computer worid, the purpose, action, syntax analysis and i emphasis on 
patterns received the most praise. The length of some lessons, and the difficulty of integrating 
these long, rather rigid lessons smoothly into their nomnal course wore most often cited as 
negative features. Many teachers indicated that they wanted more freedom than the tightly 
controlled conditions of the experiment permitted. It was apparent that infusing new concepts 
ir'o their normal lessons is a skill not quickly mastered, and while we woukl expect easier 
integration Ihe second time around," future interventions need to address these concerns. 
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Year End Follow-up 

A number of our experimental and control groups happened to be year-tong BASIC 
classes, which enabled us to collect some evidence on the longer term effects of the 
Metacourse on competence in BASIC Programming. As indicated in Table 4 we were able to 
retest a subset of students from each group in Juno of that year on an equivalent BAS'C test 
(identical problems with numbers changed). The results of this analysis must be interpreted 
with caution since both the experimental and control groups were selected sub-sanrples of 
the original groups. In particular, all seniors had already left along with those students who 
had been in the one-semester BASIC courses. The experimental sub-group, and 
particularly the available control sub-group, were superior in performance to the original 
experimental and control groups - the experimental subset made about one less error then 
the original experimental group, while the control subset made neariy five less errors than 
their original counterparts (see Tables 1 and 4). 

With these cautions in mind, note that the experimental sub-group maintained their 
performance over the year with roughly the same number of errors in June as in January. 
The control sub-group actually improved their performance over this time by about two fewer 
enfors. This enhanced performance may be attributed to the fact that after the first 
semester's testing the control teachers were sent all the Metacourse materials and told they 
would be free to use them as they wished now that the f omnal experiment was over. Thus ma 
ny of these students may well have been exposed to Metacourse concepts during the 
second half of the year. The fact that the control sub-group was a much superior sub-set of 
the original group, and could be expected to continue to improve throughout the year, may 
be a further factor. 

Discussion 

The large scale study reported here was undertalcen in order to assess the effects of 
a "metacourse" which was designed to enhance novices' leaming of BASIC by providing a 
small number of key concepts and strategies that, our clinical research revealed, mary 
students fail to acquire on their own (Peri<ins, Hancock, Hobbs, Martin, & Simmons, 1986; 
Pericins & Martin, 1986: Peridns, Martin , & Farady, 1986: Peridns, Farady, Hancock, Hobbs, 
Simmons, Tuck, & Villa, 1986). A pitot study reported in Peridns, Farady, Simmons & Villa 
(1986), employing an eariier version of the Metacourse, although encouraging, produced 
little hard evklence on its effectiveness. It was hoped that the current study would be able to 
provkle stronger evidence on the issue. 
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TABLE 3. Errors (Mean Number) on Problems In the cognitive Pre- 
and Post-tests. 



Problem Type 



Condrtional Reasoning Visual Planning 





Pretest Posttest 


Gain 


Pretest 


Posttest 


Qam 


Exp. 


1.71 1.56 


.15 


6.24 


4.92 


1.32 


Cont. 


1.36 1.33 


.03 


5.26 


4.81 


.45 


Diff. 


.35" .23 


.12 


.98 


.11 


.87 






Prohlem Tvoe 








Professors/Students 


Precise Visual Descriotion 




Pretest Posttest 


Gain 


Pretest 


Posttest 


Gain 


Exp. 


.72 .76 


-.04 


2.87 


2.63 


.24 


Cont. 


.68 .54 


.14 


2.02 


2.03 


-.01 


Diff. 


.04 .'c2"* 


-.18* 


.85** 


.60* 


.25 






Problem Tvoe 








Reoeats and necid^s 




Total 






Pretest Posttest 


Gam 


Pretest 


Posttest 


Gain 


Exp. 


4.84 3.40 


1.44 


16.34 


13.22 


3.12 


Cont. 


2.95 3.53 


-.58 


12.34 


12.29 


0.05 


Diff. 


1.89"* -0.13 


2.02*** 


4.00" 


.93 


3.07*" 



Experimental Group (n-122) 
Control Group (n>219) 



* p<.05 
" p<.01 
'" p<.001 
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TABLE 4. Errors (Mean Number) on the BASIC Test of Experimental and 
Control Students In Year-Long Courses. 



January Test Juna Test Difference 

Exp 13.30 13.10 .20 

Cent. 15.00 13.00 2.00" 



Experimental Sub-Group (n-60) * p<.05 

Control Sub-Group (n-43) p<.01 
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As in our previous report it is appropriate to consider the three questions: Did the 
Metacourse prove ^Yeachable", leading to an implementation that snfioothly provided the intended 
concepts and practice in their use? Did the Metacourse have the hoped-for impact on students' 
1 mastery of programming? Did the Metacourse, or the normal programming instruction in the 

control groups, have a cognitive impact beyond the targeted instruction? Finally questions of 
possible long term effects, and future directions should also be briefly considered. 

Teaching 

Once again the data indicate that at least for experienced teachers under conditions of 
considerable support the Metacourse was quite teachable and could be integrated into the 
normal cunlculum in BASIC. Although the Metacourse lessons produced a variety of differences 
in teacher style of presentation as well as student behavior, compared to the nonnal lessons, both 
teachers and students appeared to adapt well to the new material with a number of indicators 
pointing to effective classes. Further it may well be the case that some of these differences in 
obsen/ed behaviors (e.g., amount of student-teacher interaction) might be due to the fact that 
this was the first time the teachers used the materials, and that such differences may diminish as 
teachers get more familiar with the material. 

The evidence on the extent to which the instructors were able this first time through to 
"infuse" the key Metacourse concepts into their entire course is not as clear. We did find that 
during the Metacourse lessons teachers referred to eariier key concepts quite often, but the 
results are not clear from the limited number of non*Metacourse lessons we could observe as to 
how often these concepts were used in regular classes. We did note a gerieral absence of 
bridges or applications of these concepts to any areas outside of the programming problems 
themselves. It seems clear that deliberate efforts have to bt» orchestrated if such bridges are to be 
built. 

Transfer 

Although our results were in general consistent with most of the literature, in that there was 
little evkjence of transfer either on most cognitive tasks or on our affective measure, we did obtain 
evidence of transfer on one subtest that was structually somewhat similar to a typical programming 
task. It may also be the case that an affective instrument more specifically focused on students 
attitudes conceming errors may yet reveal some changes in this realm. 

As we have pointed out prevfously, we believe a kind of tradeoff may exist between teaching 
for programming competency and teaching for transfer of cognitive skills from programming 
(Peri<ins, Schwartz,& S!mmons,in press). The metacourse as currently designed still focuses on 
the development of programming competency rather than transfer of general cognitive skills. We 
provide little that does not have direct bearing on programming competency. 

Irrpact on programmir^ mastery 

It is clear that our experimental groups evkjenced a considerable advantage <n their general 
competence as beginning BASIC programmers compared to the control groups. Further, the 
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results indicate that this improvement is manifested on a variety of programming sl^ills, such as: 
use of correct syntax to perform simple operations, ability to trace the actions of a program through 
hand execution, and the ability to debug and produce simple programs. The evidence also 
suggests that this mastery of elementary BASIC skills does not deteriorate over a 5 month period 
after the formal intervention had been completed. While the Metacourse did not produce 
enhanced *)erformance on every problem, e.g., detecting the 'parallelism bug" (Pea,1985), it did 
produce fewer enrors on a variety of significant problems typical of those encountered in an 
introductory BASIC course. What is not so apparent is the cause of the improved performance. A 
number of possibilities are plausible, and the results may well be due to a combination of factors. 

We would hope that the Metacourse with its emphasis on the students' development of 
mental models through which they understand what the computer does, strategies by which to 
organize their problem-solving efforts, etc. played a significant role. However, it must be 
acknowledged that the "laboratory sites" intervention of whtoh this study was a part was an atypical 
treatment in that the participating teachers received much wore support than is normally the case 
when a new cur.lculum intervention is introduced. Further, we couki not randomly assign 
teachers to treatments. The laboratory site high school programming teachers were all required to 
use the Metacourse, while the control group was formed by soliciting volunteer teachers at similar 
high schools. While the teachers in both groups were experienced teachers of BASIC, it is 
possible that our treatment teachers were simply a group of exceptionally talented instructors who 
couki produce large gains even with a treatment with which they were unfamiliar. One should 
note that our study, like most in the field, pits teachers using a treatment program for the first time 
against controls who emptoy their "normal" curriculum which they have typically worthed through 
many times. A second study was therefore designed to help resolve soma of these ambiguities. 



STUDY 2 

Method 

In this study the revised Metacourse was us^d by 9 other teachers in 7 n^w high schools 
under conditions that nfx>re neariy duplicate normal classroom innovations. That is, these 
teachers were given the Metacourse atong with some "Metacourse memos" (notes to teachers on 
experiences teaching with the Metacourse), and are provkied with no other supports. No 
observations were made of these classes . The same assessment instruments were emptoyed as 
in the previous study. It was hoped that res^jlts from this study wouM help delineate the 
significant factors influencing the improved performance in BASIC. 
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TABLE 5. Errors (Mean Number) on Each Problem of the BASIC Test for 
Study 2. 



Simple One Line Command Problems 





1 


2 


3 


4 


5 


6 


7 


8 


9 


Total 


Exp. 


.10 


.17 


.36 


.11 


.25 


.74 


.76 


.78 


.26 


3.53 


Cent. 


.08 


.11 


.72 


.16 


.22 


1.14 


.88 


1.13 


.49 


4.92 


DIff. 


.03 


.06 


-.36"* 


-.04 


.03 


-.40*" 


-.12 


-.35** 


-.23*** 


-1.39" 



D/S.D .36 



Hand FYecution Problems DeBuqpina Problem 





10 


11 


13 


Total 


12 


Exp. 


.41 


.97 


1.72 


3.10 


.84 


Cent. 


.73 


1.47 


1.94 


4.14 


1.05 


Diff. 


-.32*** 


-.50*" 


-.22 


-1 .04*** 


-.21 


D/S.D 






.43 




.22 



Prod-irtinn Prnhlemfi All Problems 





14 


15 


16 


Total 


Total 


Exp. 


1.53 


2.29 


3.83 


7.66 


15.12 


Cent. 


1.60 


3.11 


5.07 


9.77 


19.88 


DIff. 


-.07 


-.81* 


-1.23** 


-2.11* 


-4.76*** 


D/S.D 








.29 


.39 



Experimental Group (n-120) 
Control Group (na224) 



• p<.05 
** p<.01 
" p<.001 
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Proportion 
of 
Errors 





Kav 


□ 


Ejf^ Lab 




Expf Nam 


■ 


Control 




One Line Hand Execution Debugging Production All Problems 

Command 



Major Problem Types 



Hgura 1: Dlffarancos In Errors (Moan Proportion) of Ono Control and Two 
Mataooursa Groupa on Major Probiam Typas in tha BASIC TasL 
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TABLE 6. Differences In Errors (Mean Proportion) of One Control and Two 
Metacourse Groups on Major Problem Types In the BASIC Test. 



Differences 

1) Exp.- Lab 

2) Exp.- Norm 

3) Control 



Problem Type 



One Line Command Prnhlems 

Prop. Errors Exd.- cnntmi (Dm\ 



.18 
.18 
.25 



1 vs3. 
2vs3' 



.or" 
.or* 



Hand Execution Prnhlems 

Prop. Errors Exp.- Control 



.21 
.24 
.32 



1 vs3-.11*** 

2 vs 3 » .08*** 



1) Exp.- Lab 

2) Exp. -Norm 

3) Control 



Problem Tvoe 



Debugging Prnhlem 



Prop, Errors 

.20 
.28 
.35 



ExD.-Contml miff ^ 

1 VS3-.15*** 

2 VS 3 - .or 



Production Prnblems 



Prop. Errors 
.29 
.31 
.39 



Exp.- Control f Diff.^ 

1 vs3 = .10*** 

2 vs 3 - .08** 



Problem Type 



1) Exp.- Lab 

2) Exp.- Norm 

3) Control 



Prop-Error 

.23 
.25 
.33 



All Problems 



Exp - Control fPiff.^ 
1 VS3-.10*** 
2vs3«.08*** 



Experimental Lab Group (n « 120) 
Experimental Nonnal Classroom Group (n-l20) 
Control Group (n-219) 



p<.05 
p<.01 
p<.001 
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TABLE 7. Analysis of Variance of Errors Made by Treatment and Control 
Group on Problem Types In the BASIC Test. 



SOURCE SS DF MS 



Bfltwftftn Groupi? 

Treatment Groups 745.6 2 372.8 10.3**' 

Subjects within groups 15711.3 433 36.3 
Within firnups 

Problem Type 11136 1 3 3712.0 

Treat. X Prob. Type 213.9 6 35.7 3.0** 

Prob. X Sub. within Groups 15258.9 1299 11.8 



* p<.05 

* p<.Ol 
" p<.00l 
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TABLE 8. Types of Errors (Mean Number) on the BASIC Test 



Simple One Line fjommand Errors rProblems 1-9^ 

123456789 

PRINT QUOTE LEI ASSIGN action goto input miSC. total 

Exp. .09 .14 .48 .63 .80 .26 .20 .93 3.53 
Cent. .05 .22 .80 .72 .93 .49 .35 1.37 4.92 
Diff. .04 -.08 -.33*" -.09 -.13 -.23"* -.15" -.44* -1.39" 



Hand Execution Errors ^Prohiftms 10 11.13^ 





9 

prompt-input 


10 11 
OUTPUT MINOR EX. 


12 
FORMAT 




13 
TOTAL 


Exp. 


1.37 


.06 .81 


.00 




2.25 


Cent. 


1.87 


.08 1.17 


.06 




3.17 


Diff. 


-.49* 


-.02 -.36" 


-.06 




-.92*** 




Debuoaina Errors fProblems 11.12.13^ 








13 
BUQ-11 


14 15 
BUQ-12 BUQ-13 


TOTAL 




Exp. 


.56 


.30 .84 


1.69 






Cent. 


.76 


.48 .78 


2.02 






Diff. 


-.20"* 


-.19" .06 


-.33* 


* 






Production Errors fProblems 14.15.16) 








16 17 
COUNT, action 


18 19 20 21 22 23 
FOR NEXT VAR. E)CTRA. MIN .<5YN. MISC. 


TOTAL 


Exp. 


.50 .49 


.44 .62 .68 1.03 


.56 


3.35 


7.66 


Cent. 


.58 .57 


.64 .76 .74 1.35 


1.16 


3.98 


9.77 


Diff. 


-.09 -.08 


-.20* -.14 -.06 -.32* 


-.60"* 


-.63 


-2.11* 



Experimental Group (n-1 20) * p<.05 

Control Group (n-224) " p<.01 



p<.001 
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TABLE 9. Errors (Mean Number) on Problems In the Cognitive Pre- 
and Post*tests. 



Problem Type 

Conditional Reasoning Visual Planning 





Pretest 


Posttest 


Gain 


Pretest 


Posttest 


Gain 


Exp. 


1.53 


1.41 


.12 


5.31 


4.53 


.78 


Com. 


1.36 


1.33 


.03 


5.26 


4.81 


.45 


Diff. 


.17 


.07 


.09 


.05 


-.28 


.33 



Problem Type 

Professors/Students Precise Visual Description 





Pretest 


Posttest 


Gain 


Pretest 


Posttest 


Gain 


Exp. 


.72 


.63 


.08 


1.82 


2.28 


-.47 


Cent. 


.68 


.54 


.14 


2.02 


2.03 


-.01 


Diff. 


.03 


.09 


-.06 


-.20 


.25 


-.46 



Problem Typ e 



Repeats and Decides jQiai 





Pretest 


Posttest 


Gain 


Pretest 


Posttest 


Gain 


Exp. 


3.08 


3.50 


-.42 


12.44 


12.35 


.09 


Cent. 


2.95 


3.53 


-.58 


12.34 


12.29 


.05 


Diff. 


.13 


-.03 


.16 


.10 


.06 


.04 



Experimental Group (n«122) 
Control Group (n«219) 



* p<.05 
" p<.01 
p<.001 
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Results 

The results of our second study are preserrted in Tables 5-9. which correspond to the 
analyses performed on the data from the Laboratory site study. Note the pattern of errors 
obtained in the new study (Table 5). is nearly identical to that obtained in the first study (Table 1 ) 
with the experimental group perfomning better than controls on all four major types of problems, 
with an overall advantage of about .39 of a standard deviation, compared to the .45 standard 
deviation advantage obtained previously. This pattam can be seen more clearly in Rgure 1 and in 
Table 6. where the proportion of errors of each experimental group are compared with each other 
and the control group for the four problem types, and overall. The students in the "normar 
metacourse treatments do about as well as those in the laboratory site" conditions, but in all 
cases make significantly fewer errors than the controls (Tukey 1sd protected test). As expected 
all groups make the highest proportton of errors on "productton problems", and the fewest on the 
"one line command problems". An Analysis of variance of these resjits (Table 7) confirms the 
treatment effect . as well as a statistically significant treatment group by problem type interactton 
indteating that the laboratory site groups made somewhat wore simple one line command errors 
than the normal 3xperimental group, but less of the other three types of errors. (The F for problem 
types is not of interest since each problem type had a different number of errors possible). ^ 

Likewise the finer grain analysis in terms of individual types of errors (Table 8) corrpared to 
the sanfie analysis in the original study (Table 2) . again confirms the eariier picture. The 
experimental groups were never outperformed by the controls, and did significantly better except 
in a few cases. Thus on the Debugging errors we again find the experimental group making 
signifteantly fewer errors on all problems except the parallellism bug (category 15). While the 
advantage of our experimental groups on productton errors did not in all cases reach statistical 
significance compared to the controls, the trends were identical to those found previously. 

The results obtained on the Cognitive Post-test (Table 9) did contrast somewhat with those 
obtained prevtously (Table 3). We had found a significant gain in performance in our laboratory 
site experimental groups compared with controls due primarily to a strong improvement on the 
"repeats and decides" sub-test. However, in the second study there were no significant 
differences between the experimental and control groups on any Cognitive Post-test measures. 

Discussion 

The results of our second study lend strong support to the hyothesis that the 
Metacourse In BASIC can produce signifk:ant irrprovement in the mastery of that language in 
a one semester high school course. Under conditions more neariy duplicating those found 
in "normar classrooms, we again found signiftoantly enhanced performance in our 
Metacourse groups on an end of semester BASIC test. Further the analysis of the students 
performance on the different types of problems also mirrored the pattern found in our first 
study. The BASIC Metacourse seems to be an intervention that programming teachers can 
readily adopt and incorporate into their normal instmction even without specal training and 
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support. While modest in scope, this Vitamin shot" is likely to produce significantly better 
mastery of BASIC in their students. 

The situation is quite different with respect to transfer. While we obtained some 
evidence for "near transfer (on the "repeats and decides" problem) in our laboratory site 
study, even this modest evidence of transfer washed out in our second study under more 
"nomiar classroom corxJitions. Thus we have further confirmation of the common finding 
that transfer from programming is not normally obtained unless special efforts are made 
explicitly to design such elements into the interventton. Where transfer was explicitly 
discussed and valued with our laboratory site teachers, some nrx)dest transfer was obtained, 
but not so in the "normar classroom situation without these supports. 



FUTURE DIRECTIONS 

While the results obtained in these studies are encouraging, even the perfomiance 
of the treatment groups on our BASIC end of semester test under the laboratory site 
conditions, left considerable room for improvement (77% correct responses). Further, 
teachers comments made it clear that, athough they could readily adopt the Metacourse in its 
cunrent nine lesson format, both the optimal integration into their normal curriculum, and the 
infusion of key concepts throughout the term, remains a considerable challenge. Students 
still often exhibit an inert, rather than active knowledge of the programming enterprise. This 
"fragile" knowledge does not permit optimal use in contexts when it should be applied, even 
within the discipline of programming let alone in other problem solving realms. The 
programming research group at ETC is currently devetoping a revision of the Metacourse to 
address some of these concems. 



This revision is built upon a "module" rather than a "lesson" model. Thus, rather than 
a collection of fully worked out lessons, complete with technical terms , exanrples and 
exercises, which some teachers found constraining, a small number of short modules are 
presented. These introduce the key Metacourse concepts, and focus explicitly on how such 
concepts can be practiced or "infused" throughout the semester while permitting 
considerable f reedorr^ for the teacher to retain his or her own vocabulary , favorite exercises 
and organizatton. Rnally a number of optional modules attempt to illustrate how bridges can 
be made to academic and "real life" problems, thus encouraging transfer to more general 
cognitive skills. While provkiing teachers more freedom to utilize the Metacourse in their own 
fashton, it is designed to make more explicit and hopefully available to both teachers arKJ 
students the core elements of our approach and the rationale behind them. Such akJs as an 
"animated computer worid" with an agent performing simple operations on infomiation, 
icons, metaphors, and posters are all being considered as metacognitive supports. Ttiis 
second generation language model Metacourse will be empirically tested in the near future. 



The results of two studies on the Metacoursse for BASIC lend encouragement to 
the emptoyment of the "Metacourse" format as a viable nfX)del for effective educational 
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interventions in courses In other programming languages, or for that matter, other academic 
subjects. Impro/ed student competence in a variety of such areas might be more easily 
achieved through "A^etaa^jrse" interventions consisting of visual models of a few key 
concepts, strategies, and metacognitive aids interspersed throughout a norm. i semester 
subject matter course. As mentioned previously this modest curriculum intervention is short 
enough to be accepted by nriost teachers and incorporated into their nonnal courses, while 
being substantial enough to infuse key concepts and habits throughout the course thus 
substantially enhancing learning. 

What has become more apparent as we have interacted with teachers is that many of them 
share a goal of teaching their students in some sense "V/hat computers and programming 
languages are really all abouf, 1x)w they can be used as problem-solving tools" and not just the 
syntax of a partk:ular programming language like BASIC. It may be the case that the achievement 
of this goal through the design of a second generation metacourse is also more compatible with 
the development of general cognitive skills, and that a powerful pedagogy of programming 
focusing on attention to mental models, strategies, etc. may lead to more success at achieving 
both programming competence and transfer. 
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Worksheet 5-2 
Page 2 



DIRECTIONS: (A), Circle «nd label the counting pattern. Name: 

(B) Hand<«xecute the program. (C) Change the final print '_ 

statement so that it explains the output in more explicit terms. 



Program Area 



10 PRINT "HOW MANY PEOPLE? 
20 INPUT P 
30 G = 0 
40 PRINT 
50 PRINT 
60 PRINT 
70 FOR I = 1 TO P 
80 INPUT N 

90 IF N > 3 THEN G - G + 1 
100 NEXT I 

110 PRINT "THE ANSWER IS" ; G 



PLEASE ENTER THE NUMBER OP 
SLICES OF PIZZA THAr 
EACH PERSON WANTS" 



Variables Area, 



User Input Area: 



4 

2 
1 
5 
4 



□DDixicnimmmiimxmnm 



□ 
1 
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Appendix B 

Example of Purpose, Action, and Syntax 
from Mini-Manual 
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in. ULl 

PURPOSE: You use LET in a program to make the computer save information in 
Its memory for later use. The information is stored in a variable. Think 
of a variable as a box with name. Boxes with names ending in $ hold only 
strings. Boxes without the $ hold only numbers. 



ACTION 



SCREEN DISPLAY 



SYNTAX 

LET Variable Name = Number 

for examole: 
LET X « 5 

Is it really there? 
PRINT X 

LET Variable Name = Legal Numerical Expressi on 




WORK SPACE 



5 IS stored m the X dox 



fo*^ examole: 

LET Z = 5 ♦ (1 + X) 

Is it really there? 
PRINT Z 



LET Z = Z 



Is it really there? 
PRINT Z 




1 + 5 IS 6 

5*6 get? vou to 50 

30 IS stored in the Z box 



30 - 2 = 2S 

28 15 the new value for Z. 
which 12 stored in the Z box 



SIZE IS stored in the A$ 
box. 

/ IS stored in the B* box. 



LET String Variable Name « String 
for examole: 
LET A$ « "SIZE" 

LET B$ » 

Are they really there? 
PRINT AS 

PRINT BS 



Note: You can leave off tne LET statement, and get the tame result, 
for example: 

X " 2 2 is stored in tne X box» 




FRir 



Dt • •DAYS'' 



DAYS is itorad in tti» 

Dt box ■ 'i^A-'d 



Appendix C 



Example of a "Pattern" from 
the Mini-Manual 



PATTERNS 



Summing Pattern 



Purpose: To find the sum of a series of numbers 



Structure: 
Lett-0 

<begin 1oop> 

LETT = T + <numbertoadd> 
<end loop> 



Example: 

10 REM ADDS 5 NUMBERS 
20 LETT'O 

30 FOR I = 1 TO 5 

40 PRINT "ENTER A NUMBER" 

50 INPUT N 

60 LETT = T^N 

70 NEXT I 



Action: The summing variable (in this case, T) is first set to 0. Each time 
the loop body is executed, another number is added to the summing variable. 
When the loop is exited, the summing variable contains the total.- 



Counting Pattern 

Purpose: To count the number of times something is true 
Structure: Example: 

10 REM COUNTS A'S 
LET C = 0 20 LET C = 0 

<begjn 1oop> 30 FOR I » 1 TO 5 

40 PRINT "ENTER A GRADE" 
50 INPUT G$ 

IF <cond1t1on> THEN LET C = C + 1 60 IF G$ = "A" THEN LET C = C + 1 
<encl *loop> 70 NEXT I 

Action: The counting variable (in this case, C) is first set to 0. Each time 
the loop body is executed, you test a condition. If that condition is true, the 
counting variable is increased by 1. 



Appendix D 
Cognitive Skills PreVPost-Test 



Version A 

Name: . Date: 



CEMERAL IMSTRUCTIOMS 

Our clasB has been chosen to contribute to some ver/ important educational 
research. A group of people at Harvard University is attempting to discover 
the best techniques of teaching BASIC programming so that it will be more 
interesting and understandable to students. 

This post-test, given now at the end of the course, will help them to 
determine the differences among all of you who will be participating. 

The Harvard Group wants you to know how much they appreciate your help and 
they are looking forward to your comments and the wonderful information 
they will gather through your efforts. 

This is a 40-minute test comprised of a short questionnaire and a number 
of problems. 

Do not start the test until your teacher says "go". 

You may work through the test as quickly as you want. After finishing one 
part, go on to the next. 

However, we want to be sure you try all parts. So after the time for a 
particular section is up, the teacher will say, *'please go ahead to section 
A, B, C, D, E, or F (whatever it is) now if you haven't already". 

You can go back to work on a previous section If you have extra time. 

Ask any questions you have now. 



A. CHCOSE ONE ANSWER 



5 Minutes 



Pick the answer that best describes what happens to you or how you feel. 
There are no right or wrong answers* 



1. When you read a story and can't remember much of it. is it usually 
. a. because the story wasn't well written, or 

b* because you weren't interested in the story? 

2. If a teacher says to you, "Your work is fine**, is it 

^a. something teachers usually say to encourage pupils, or 

b. because you did a good Job? 

3. Suppose you weren't sure about the answer to a question your teacher 
asked you, but your answer turned out to be right. Is it likely to 
happen 

a. because she wasn't p particular as usual, or 

^b. because you gave the best answer you could think of? 

4. When you read a story and remember most of it, is it usually 
^a. because you were interested in the story, or 

b. because the story was well written? 

5. If the teacher didn't pass you to the next grade, would it probably be 
^a. because she **had it in for you", or 

b. because your school wo 'k wasn't good enough? 

6. Suppose you don't do as well as usual in a subject at school. Vould 
this probably happen 

a. because you arenH as careful as usual, or 

b. because somebody bothered you and kept you from working? 

7. If a boy or a girl tells you t^at you are bright, is it usually 
^a. because you thought up a good idea, or 

b. because they like you? 

8. Suppose you became a famous teacher i scientist or doctor. Do you 
think this would hcppen 

a. because other people helped you when you needed it, or 

b. because you worked hard. 

9. Suppose you are showing a friend how to play a game and he has trouble 
with it. Vould that happen 

a. because he wasn't able to understand how to play, or 

b. because you couldn't explain it well? 

10. If you can't work a puzzle, is it more likely to happen 

a. because you are not especially good at working puzzles, or 

b. because the instructions weren't written clearly enough? 



DAYS OF THE MEEK 



6 Minutes 



1. the day before the day after tomorrow is Sunday, what is the day 
before yesterday? . 



2. Suppose a Meek had no Wednesday. If today is Tuesday, the day after 
tomorrow is what? . 



3. Suppose today is the day before Tuesday. Mnat is the day before yesterday'' 



A. If tomorrow were Wednesday instead of Sunday, yesterday would be Monday 
instead of what? . 



G. VISUM, PUZZLE 5 Minutes 
How many triangles dors the figure below contain'^ • 



For every 3 people who drink coffee, 1 person drinks tea. Suppose C 
stands for the number of people >#io drink coffee. T stanos for the numbrr 
of people who drink tea. Circle the equation that states the rclati "unship 
between how i»any peit:<le drink coffee and how many people dr«nk tea. 




D. RmJLA QUESTION 



A Minutes 
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1 ■ C/T 



T - 3C 



C ■ 3T 



3 - C - T 



E. DESCRIPTION 



8 Hinutss 



Ex«»ln» carefully the picture* of th« item* shown in the following figure. 
Write • description of Item "G" to «omeone, «o thet he/«he could pick out 
"G" from ell the other shapes. The other person's psper dOMn't have the 
letters beside to identify etch figure, snd the figures are all mixed up. 
Write out your descriptions of "G" below. 



A. 




C. 



t 







+ 










































|1 D D J 






















O 3 0 0 






G. 





I. 




J. 
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F« REPEATS AND DECIDES 



10 HinutfeS 



For the next problem , you have to learn about repeats and decides before 
you do the problems. 



REPEAT; A repeat is an instruction to do something over and over again , 
until some condition is met. For instance: 

REPEAT - Jump up and down until you've done it 17 times • 



pECIDEi A decide is an instruction to choose which one, of two or more 
conditions exist, and then doing something once, based on this choice. 
For instance: 

DECIDE - Decide if you have homework. If you do, stay home and 
do it. If you don't, then go see a friend. 



WRITING DIRECTIONS USING REPEATS AND DECIDES; You can write directii 
using repeats and decides. Here is on example that uses one repeat and 
one decide instruction. 



Suppose you want to add up all the money in everybody's pocket in the room 
and shout "Hurrah!" if the total is over $100. 
Here are the directions: 

REPEAT - Add on the money in the next person's pocket until you 
have covered all the people in the room. 

DECIDE - DeciQv-* if you have more than $100. If you do, shout, 
-Hurrah!". If you don't, then be quiet. 



Other problems may require more than one repeat and/or decide. 
Now go to the next page for the problem. 



It'e Saturday and you w&nt to go c xt and r^e the movie called: **The Crazy 
Computer-. You have a newspaper that lists the movie theaters alphabetically 
with what is playing at each. If you can't find the movie anywhere then 
turn on the T.V. to a ball game. If you find the movie then start calling 
your frit^nds until one says he or she will go with you. If you find one 
willing to go, then make a date and go. If not stay home and watch the 
ball game. 



If you have extra time, go back and complete sections you did not finish 
or check your work. 



Appendix E 
BASIC Test 



NAME 



DATE 



DIRECTIONS — Work quickly but carefully on the following problems. If 
you get stuck on one problem, go on to the next. 



Urite the Basic commands that will cause the following to happen. 
An example is done for you below. 

EXAMPLE. The following .ord appears on the screen: 



1. The following message is to be printed on the screen: 

The answer is cat 

2. The number 5 is to appear on the screen. 

3. The program asks the user for a number. (The number will be stored in 
a variable.) 

4. The variable X is set to the value 2. 

5. The variable X is set to 3 more than Y. 

6. A variable (give it whatever name you like) is set to the following: 

THE ANSWER IS KNIGHT 



Hello 
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7. Uhen the value of Z is greater than 3, the following message is 
printed: L = 5 



Complete the next 2 programs by filling in the blanks with the appropriate 
code. 



8. This program uses a FOR statement to print the number 6 ten times. 
(Fill in the blanks.) 

10 FOR X = 1 TO 

20 PRINT 

30 

9. The following is part of a larger program. What is needed in this 
part of the program so that lines 220-230 are not executed? (Fill in the 
blank. ) 

210 

220 Print "No" 
230 Print "No" 
240 Print "Yes" 



10. During a run of the following program what will appear on the screen? 
Suppose the user enters 16 for A. 

10 PRINT "HOW OLD ARE YOU?" screen display 

20 INPUT A I , 

30 IF A=15 THEN PRINT "TOO OLD" 

40 IF A>15 THEN GOTO 60 

50 IF A<1j then print "LITTLE KID" 

60 PRINT "RETIRE FROM SCHOOL" 

70 END 



11. During a run of the above program what will appear on the screen? 
Suppose the user enters 14 for A. 

screen display 



12. This program asks the user to enter the t. umber of slices of bread 
he/she has eaten and then shows on the screen the total number of calories 
the bread contains. In this program S stands for number of slices of 
br *adi and C stands for calories. (Assume there are 70 calori'<»s in a slice 
of bread.) 

The program doesn^'t work. The output is not correct; it is always: 

Number of slices 

Total calories in bread 

0 



What is wrong with the program? Find the error (s) and correct the program. 



100 PRINT -Number of slices" 

110 LET C = S * 70 

120 PRINT -Total Calories in bread 

130 PRINT C 

140 END 



13. The following program was written to calculate weekly pay by multiplying 
hours worked times hourly wage. P stands for pay, H for hours, and U for 
wage. The user of this program types in 10 for hours and 4 for wage. 
During the run what will ^* screen display show? 

10 LET P = h * U screen display 



20 INPUT "HOURS H 

30 INPUT -WAGE W 

40 PRINT -PAY = P 

50 END 



3 
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You will be writing 2 programs that do the same thingj they print out on 
the screen the whole numbers from 1 to 10. The ou*-put will look like this: 



1 
2 
3 
4 
5 
6 
7 
8 
9 

10 



In your first program, use a FOR/NEXT loop. In your second program create 
the loop using an IF statement. 



14. PROGRAM WITH FOR/NEXT LOOP 15. PROGRAM WITH IF STATEMENT 



4 



16. Each day, a grade school class ot 20 students counts the number of 
cartons of milk needed by the class. Write a program that will: 1) for 
each of the 20 students ask how many cartons of milk he or she will drink, 
and 2) calculate the total number of cartons needed by the class and print 
the total on the screen. 



Appendix F 
Classroom Observation Sheet 



OBSERVATION INSTRUMENT 

Observer! Date: 

School: 

Teacher: 



Beginning time: . ending time: 

total class time: 

control: 



experimental: metacourse lessor. # 

ordinary class 

taped? yes no 



1. Write out an outline of the class as things happen. Try to note the 
essential features such as major points made, method of presentation, 
practice time, etc. Revise after class if necessary. 
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Instruction: Pease describe the behaviours indicated below on a 7 point 
scale. For most examples, there are definitions and reference points 
given. Feel free to assign ratings between the reference points, but use 
whole numbers only (no fractions or decimal points) . If a question is 
unanswerable or somehow not applicable, please mark an "x" in the space. 
(Where fitting, a brief explanation would be appreciated). 



2. Amount of interaction between teacher and students - 

7 - lots of interaction between students and teacher 

4 - average amount of interaction 

1 - little interaction 

3. Amount of broadly beneficial interaction among students - 

7 - much 

4 - some; a few comments between students 

1 - none 



4. Amount of deleterious interaction among students 

7 - much 

4 - some; the usual amount of fooling around 

1 - none 



5. Student responsiveness - 

7 - students respond readily 

4 - students need to be coaxed 

1 - students are unwilling to participate, to answer or to 

ask questions, etc. 



6* Attentiveness of students - 

7 - highly attentive to material/instruction 

4 - somewhat attentive, but also talking, etc 

1 - inattentive — talking, doing other work 



Percentage of students who seem to be engaged in class activities 



8. Teacher - student interaction ( Give percentages) 



Teacher's role Students' role 

a. Process oriented 

b. Product oriented 

c. Disciplinary 



9. Preparedness of students - 

7 - students seem very well prepared and ready to move on 

4 - students seem adequately prepared 

1 - students seem to lack understanding requisite to 

continuing 

10. Write in the approximate percentages of the total time 

spent in: 



lecture 

questions and discussion 

demonstration 

hand execution exercise 

other exercises 



(total should be 100%) 



lecture - teacher speaks 

demonstration - teacher demonstrates some aspect of programming 

either on board or with overhead 
hand-execution - students/teacher go over program line by line 
other exercises- . 



11. Teacher-material interface - 

7 - teacher seems very comfortable and conversant with 

material 

4 - teacher seems adequately comfortable with material 

1 - teacher seems uncomfortable with material 

12. Mode of instruction 



Give percentages. 

- highly interactive; teacher often asks for comments, 

talks to and with students 

- combination of interactive and presentational 

- presentational; teacher describes, demonstrates, 

models 



4 



13. Subjective, wolistic rating of total effectiveness of 

instruction 

i3a. Comipcnt - 



14. Reference to other lessons - 

7 - much reference to earlier lesson(s) 

4 - some mention of earlier ^«i5Sons(s) 

1 - no mention of earlier lesson(s) 



14a. If ideas from earlier lesson(s) were referred to, note whicn 



15. Check: (use 1 for "yes" or 2 for '^no") Were there any bridges or 
applications stressed for which Basic Programii>ir.g princijilas are used in: 
a. other academic disciplines; which one(s): 



b. problems in real life, outside school; whict one(s): 



How? Please note: 



For experimental lesson classes only 
Score using the following scale: 

7: gfuded use, using it a lot 

4: a propric>te, right on target 

1: no mention 

16. Which basic principles from the Metacourse were referrec to: 



i the paper computer 

b. purpose, syntax, action 

c. _ interaction between the user and the computer 

d. the minimanual 

e. patterns 

f. program production using metacourse principles 

g. other: 




17. rcentage of the important material in this lesson 

that was adequately covered* 



Fidelity of lesson observed to lesson as written - 
7: taught almost exactly as written 

4: the same or very similai content of lesson as written* but adapted 
and paraphrase'.' 

1^ almost unrecognizable as the same lesson 

18. Introduction 

19. Lecture 

20. Participation 

21. Explanation 

22. Exercise 



Appendix G 
Student Questionaire 



'■1 



Previous Cannputinq Experience 



NAME 



Please answer the following questions concerning your experience with 
computers. If you have a hard time answering any of these questions just 
give it your best guess. The informatJon will help us in our research, 
and we appreciate your help with this project. 



1) Have you ever taken a BASIC programming course before this semester? 
Yes No 



2) If you answered i^es to number 1: When did you take the cour5e(es), how 
long did the course(s) last and how often did the class(es) meet? (For 
example, you ....jht have had a BASIC course at a camp this past summer that 
lasted for 6 weeks and met 3 times a week for 2 hours each class.) 



3) Have you ever taken another progran^ning course in a language other than 
BASIC? 

Yes No 

4) If you answered yes to question 2: What language(5) have you studied? 

Logo If yes, how much? 

Pascal If yes, how much? 

Other (name of language) If yes, how much? 

5) In what grade were you first introduced to computer languages (Logo, 
Basic, etc)? 

6) In what grade were you first introduced to computer apol i cat ions (word- 
processing data bases, spreadsheets)? 

7) Is there a computer in your home? If so, 1) what kind is it, 2) how often 
do you use it, and 3) what 2 things do you use it for the most (for example, 
games, programming, wordprocessing)? 



